A GENERAL FORM OF GELFAND-KAZHDAN 

CRITERION 



BINYONG SUN AND CHEN-BO ZHU 

Abstract. We formalize the notion of matrix coefficients for dis- 
tributional vectors in a representation of a real reductive group, 
which consist of generalized functions on the group. As an appli- 
cation, we state and prove a Gelfand-Kazhdan criterion for a real 
reductive group in very general settings. 



1. Tempered generalized functions and representations 

in the class j^h 

In this section, we review some basic terminologies in representation 
theory, which are necessary for this article. The two main ones are 
tempered generalized functions and representations in the class J^l-i. 
We refer the readers to |W88t IW92] as general references. 

Let G be a real reductive Lie group, by which we mean that 

(a) the Lie algebra g of G is reductive; 

(b) G has finitely many connected components; and 

(c) the connected Lie subgroup of G with Lie algebra [g, g] has a 
finite center. 

We say that a (complex valued) function / on G is of moderate 
growth if there is a continuous group homomorphism 

p : G — J- GL„(C), for some n > 1, 

such that 



< tr(p(a;) p{x)) +tr(p(a;-i) p{x'^)), xeG. 

Here stands for the complex conjugation, and " *" the transpose, 
of a matrix. A smooth function / G C°°(G) is said to be tempered if 
Xf has moderate growth for all X in the universal enveloping algebra 
U(gc). Here and as usual, gc is the complexification of g, and U(0c) 
is identified with the space of all left invariant differential operators on 
G. Denote by C^(G) the space of all tempered functions on G. 
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A smooth function / G C°°{G) is called Schwartz if 
\f\x,^ := sup^.gG0(a;) \{Xf){x)\ < oo 

for all X G U(gc), and all positive functions on G of moderate 
growth. Denote by C'^(G) the space of Schwartz functions on G, which 
is a nuclear Frechet space under the seminorms {| -Ix.^}- (See |Tr67] 
or |Ta95] for the notion as well as basic properties of nuclear Frechet 
spaces.) We define the nuclear Frechet space D'^{G) of Schwartz den- 
sities on G similarly. Fix a Haar measure dg on G, then the map 

C'{G) D'{G) 
f ^ fdg 

is a topological linear isomorphism. We define a tempered generalized 
function on G to be a continuous linear functional on D^{G). De- 
note by C^^(G) the space of all tempered generalized functions on G, 
equipped with the strong dual topology. This topology coincides with 
the topology of uniform convergence on compact subsets of D'^(G), due 
to the fact that every bounded subset of a complete nuclear space is 
relatively compact. Note that C^(G) is canonically identified with a 
dense subspace of C~^{G): 

Remark 1.1. In |W92] . the space C''{G) is denoted by S{G) and is 
called the space of rapidly decreasing functions on G. Note that C''(G) 
(or S{G)) is different from Harish- Chandra's Schwartz space of G, 
which is traditionally denoted by C{G). 

By a representation of G, or just a representation when G is un- 
derstood, we mean a continuous linear action of G on a complete, 
locally convex, Hausdorff, complex topological vector space. When no 
confusion is possible, we do not distinguish a representation with its 
underlying space. Let \^ be a representation. It is said to be smooth if 
the action map G x V ^ V is smooth as a map of infinite dimensional 
manifolds. The notion of smooth maps in infinite dimensional setting 
may be found in |GN09j . for example. 

Denote by C(G; V) the space of V^-valued continuous functions on 
G. It is a complete locally convex space under the topology of uni- 
form convergence on compact sets. Similarly, denote by C°°(G; V) the 
(complete locally convex) space of smooth V^-valued functions, with the 
usual smooth topology. For any v E V, define G C(G; V) by 

Cvig) ■■= gv. 
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The vector v is called smooth if G C^{G]V). Denote by V^o the 
space of all smooth vectors in V , which clearly is stable under G. Note 
that the linear map 

is injective with closed image. Equip V^o with the subspace topology 
of C°°{G\V), then it becomes a smooth representation of G, which 
is called the smoothing of V . If V is smooth, then V^o = as a 
representation of G. In this case, its differential is defined to be the 
continuous U(gc) action given by 

Xv = {Xc,){l), XgU(0c), veV, 

where 1 is the identity element of G. 

The representation V is said to be Z(gc) finite if a finite codimen- 
sional ideal of Z(0c) annihilates Voo, where Z(gc) is the center of U(gc). 
It is said to be admissible if every irreducible representation of a max- 
imal compact subgroup K oi G has finite multiplicity in V . A rep- 
resentation of G which is both admissible and Z(gc) finite is called a 
Harish- Chandra representation. 

The representation V is said to be of moderate growth if for every 
continuous seminorm | |^ on V, there is a positive function on G of 
moderate growth, and a continuous seminorm | on such that 

Ifl'^^U — 4'{.9)\''^\ui for all (7 G G, f G V. 

The representation V is said to be in the class if the space V is 
Frechet, and the representation is smooth and of moderate growth, and 
Harish-Chandra. The category of all such V is denoted by the same 
symbol (the morphisms being G-intertwining continuous linear 
maps). The strong dual of a representation in the class is again a 
representation which is smooth and Harish-Chandra. A representation 
which is isomorphic to such a strong dual is said to be in the class VH. 
By the Casselman-Wallach globalization theorem, both the category 
J^l-i and VH are equivalent to the category % of admissible finitely 
generated (gc, ^)-modules f fUSQ] . [W92l Chapter 11]). 

Remark 1.2. Our terminology follows that of Wallach in [W92j . where 
presumably T refers to Frechet, "H refers to Harish-Chandra, and 
hints at Frechet globalization of Harish-Chandra modules. Note that 
Casselman calls a representation in the class J^H a Harish-Chandra 
representation ( [C89\ . page 394), while some other authors spell out all 
conditions of the class J-'Ti, without attaching a name. 
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From the theory of real Jacquet modules (by Casselman and Wal- 
lach), and the Casselman- Wallach globalization theorem, every repre- 
sentation in the class J^l-L is the smoothing of a Hilbert representation. 
In addition, all representation spaces in Tl-L are automatically nuclear 
Frechet (and in particular are reflexive), and all morphisms in J^l-L and 
VH are automatically topological homomorphisms with closed image. 
See |W92t Chapter 11]. Recall that in general, a linear map \ : E F 
of topological vector spaces is called a topological homomorphism if 
the induced linear isomorphism i?/Ker(A) — )■ Im(A) is a topological 
isomorphism, where £'/Ker(A) is equipped with the quotient topology 
of E, and the image Im(A) is equipped with the subspace topology of 
F. 

2. Statement of results 

Let U°°, V°° be a pair of representations of G in the class which 
are contragredient to each other, i.e., we are given a G-invariant non- 
degenerate continuous bilinear map 

(1) (, ) : f/°° X ^ C. 

Denote by U~°° the strong dual of V°°, which is a representation in 
the class VH containing f/°° as a dense subspace. Similarly, denote by 
V~°° the strong dual of U°°. For any u G U°°, v G V^, the (usual) 
matrix coefficient Cu^^ is defined by 

(2) Cu^y{g) := {gu,v), geG. 

By the moderate growth conditions of U°° and V°°, one easily checks 
that a matrix coefficient Cu(^v is a tempered function on G. 

The following theorem, which defines the notion of matrix coefficients 
for distributional vectors, is in a sense well-known. See the work of 
Shilika ( |S74l Section 3]) in the context of unitary representations, and 
the work of Kostant ( |K78[ Section 6.1]) or Yamashita ( [Y881 Section 
2.3]) in the context of Hilbert representations. With the benefit of the 
Casselman- Wallach theorem, it is of interest and most natural to state 
the result in the context of representations in the class J^H. This is also 
partially justified by the increasing use of this class of representations, 
due to the recent progress in restriction problems for classical groups. 
One purpose of this note is to provide a detailed proof of this result. 

Theorem 2.1. LetG be a real reductive group. Denote by C^{G) (resp. 
C^^{G)) the space of all tempered functions (resp. tempered generalized 
functions) on G. Let (t/°^, V"°°) be a pair of representations of G in 
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the class which are contragredient to each other. Then the matrix 
coefficient map 

extends to a continuous bilinear map 

and the induced G x G intertwining continuous linear map 
(4) c : U-°°^V-^ C-^{G) 

is a topological homomorphism with closed image. 

Here "®" stands for the completed projective tensor product of Haus- 
dorff locally convex topological vector spaces. In our case, this coincides 
with the completed epsilon tensor product as the spaces involved are 
nuclear. Recall again that a linear map X : E F of topological vec- 
tor spaces is called a topological homomorphism if the induced linear 
isomorphism i?/Ker(A) — )■ Im(A) is a topological isomorphism, where 
£'/Ker(A) is equipped with the quotient topology of E, and the image 
Im(A) is equipped with the subspace topology of F. The action of 
G X G on C^^{G) is obtained by continuously extending its action on 
C«(G): 

{{9i,92)f){x) := fig^^xgi). 

Remark 2.2. (A) Denote by tjj^ the paring x f/°° — )• C, and view 
it as an element ofU'^^V-'^. Then c{tuj) G C"^(G) is the character 
of the representation U°° . 

(B) Let U^, f/|°, ■ ■ ■ , be pairwise inequivalent irreducible represen- 
tations of G in the class Let V°° be a representation of G in the 
class J^T-L which is contragredient to U°° , i = 1,2, ■■■ Then the 
second assertion of Theorem \2. 1\ implies that the sum 

k 

0f/r°°§v;-°" ^ c-^{G) 

i=l 

of the matrix coefficient maps is a topological embedding with closed 
image. 

A second purpose of this note (and additional reason for writing down 
a proof of Theorem 12. ip is to prove the following generalized form of 
the Gelfand-Kazhdan criterion. For applications towards uniqueness 
of certain degenerate Whittaker models, it is highly desirable to have 
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the most general form of the Gelfand-Kazhdan criterion. We refer the 
reader to |JSZ08j for one such apphcation. 

Theorem 2.3. Let Si and S2 be two closed subgroups of G, with con- 
tinuous characters 

X^■.S,^C\ 1 = 1,2. 

(a) Assume that there is a continuous anti- automorphism t of G 
such that for every f G C~^{G) which is an eigenvector of 
U(gc)'^, the conditions 

f{sx) = xiis)f{x), s e Si, 

and 

f{xs) = X2isy^f{x), s e S2 

imply that 

f{x^) = fix). 

Then for any pair of irreducible representations {U°°, V°°) of G 
in the class J^T-L which are contragredient to each other, one has 
that 

dimHoms,(f/~,C;,J dimHom5,(\/°°, C^J < 1. 

The equality holds only ifU?° and V°° are equivalent, where U?° 
is the representation obtained from U°° by (pre) composing with 
the automorphism g h-> T{g~^). 

(b) Assume that for every f G C^^{G) which is an eigenvector of 
11(0)"^, the conditions 

f{sx) = xiis)f{x), s e Si, 

and 

fixs) = X2isy^f{x), s e S2 

imply that 

f = o. 

Then for any pair of irreducible representations {U°°, V°°) of G 
in the class which are contragredient to each other, one has 
that 

dimHom5,(f/°°,CxJ dimHom5,(\/°°, C^J = 0. 

Here and as usual, U(gc)*^ is identified with the space of bi-invariant 
differential operators on G, C^^ is the one dimensional representation 
of Si given by the character Xi, and "Hom^." stands for continuous Si 
homomorphisms. The equalities in the theorem are to be understood 
as equahties of generalized functions. For example, f{sx) denotes the 
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left translation of / by s ^. Similar notations apply throughout this 
article. 



Remark 2.4. (A) In concrete applications, the data 5*1, 5*2, Xi? X2? 
are interconnected, and various considerations often imply that the 
two terms in the product dim Hom^j {U°°, C^^-J dim Hom^j {V°°, C^^) ^'^^ 
equal, and so each of them is at most 1. (One case is that Si = S, 
S2 = t{Si), = o f, and X2 = Xi° ^■) This is how Gelfand- 
Kazhdan criterion (Part (a) of Theorem \2.3\) is applied to prove the (lo- 
cal) uniqueness of certain models in the theory of automorphic forms. 
Part (b) serves a similar purpose in showing the disjointness of certain 
periods. 

(B) The original Gelfand-Kazhdan criterion is in |GK71j (for the non- 
archimedean case), and their idea has been very influential ever since. 
Various versions for real reductive groups have appeared in the liter- 
ature, including |S74] and |K78] (for the study of Whittaker models, 
but both implicitly) . Later works which state some versions of Gelfand- 
Kazhdan criterion explicitly include that of H. Yamashita f |Y88[ The- 
orem 2.10]j, and of A. Aizenbud, D. Gourevitch and E. Sayag (^ [AGSj 
Section 2]). The papers |S74j and jYSSj have been particularly instruc- 
tive for the current article. 

3. Proof of Theorem 12.11 

Let ([/, (,)[/) be a Hilbert space which carries a continuous repre- 
sentation of G so that its smoothing coincides with U°° . Denote by V 
the strong dual of f/, which carries a representation of G. (Its topology 
is given by the inner product 

where Ui E V is the linear functional {■ ,Ui)u on U.) Note that the 
smoothing of V coincides with V°°. Recall, as is well-known, that the 
three pairs U and V, U°° and V~°°, and U~°° and V°°, are strong duals 
of each other as representations of G. For u E U, v E V, set 

\u\u := \/ {u,u)u and \v\v := a/ {v,v)v 

Lemma 3.1. There is a continuous seminorm \-\g on D^(G) such that 

f{g)\ \gu\udg < \u\g \u\u, u = f dg E D'^(G'), u E U. 

Proof. This is well known, and follows easily from the facts that 

(a) U is (automatically) of moderate growth ( |W88l Lemma 2. A. 2. 2]), 
and 
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(b) there is a positive continuous function on G of moderate 
growth so that 1/0 is integrable. See |W88[ Lemma 2. A. 2. 4]. 

□ 

By Lemma [3.11 for any u G D'^(G') and u & U, the integral (in the 
sense of Riemann) 

(5) uu := / uj{g)gu 

Jg 

converges absolutely, and thus defines a vector in U. Furthermore, the 
bilinear map 

D%G)xU ^ U, 

is continuous. 

Lemma 3.2. For u G D'^(G) and u ^ U, we have uu G U°° . 

Proof. Denote by L the representation of G on D'^(G') by left transla- 
tions. Thus for g E G and u G D''(G'), Lg{u) is the push forward of u 
via the map 

G ^ G, 
X H- gx. 

It is routine to check that 

L: Gx D'{G) D'{G) 

is a smooth representation. For X G U(gc)5 denote by 

Lx ■■ B%G) D'{G) 

its differential. Trivially we have 

(7) c^uig) = {Lg{u))u, g eG,ueU. 

This implies that c^ju G C°^(G'; U), namely uu G f/°°. □ 

The following two lemmas are refinements of |S74| Proposition 3.2]. 

Lemma 3.3. The bilinear map 

$[/ : D^(G) xU 

{u,u) I— i- uu. 

is continuous. 

Proof. By the defining topology on f/°°, we need to show that the map 
B'{G)xU C°^(G';f/), 



is continuous. In view of the topology on C°°(G; U), this is equivalent 
to showing that the bilinear map 

D'{G)xU C{G;U), 
is continuous for all X G U(gc)- This is clearly true by observing that 

□ 

For any u G D'^(G), denote by its push forward via the map 

G ^ G, 
9 ^ 9'^- 

Applying Lemma [3.31 to V, we get a continuous bilinear map 

D'{G)xV V^, 
{u,v) UV. 

Now for any u G D'"(G'), we define the continuous linear map 

^7-°° ^ U, 

U H-T- OJU 

to be the transpose of 

V 

V (-7- W^f, 

i.e., 

(8) {uu,v) = {u.u'^v), ueU~°°,veV. 

Lemma 3.4. The bilinear map 



zs separately continuous and extends 

Proof. It is routine to check that ([9]) extends ([6]). We already know 
that ([9]) is continuous in the second variable. 

Fix u G U~°°, then the continuity of the bilinear map 

Ou : D^(G) X F ^ C, 

clearly implies the continuity of the map 

D«(G) ^ f/, 

□ 
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Lemma 3.5. The image of $y is contained in U°° , and the induced 
bilinear map 

is separately continuous. 

Proof. By chasing the definition of uu, we see that the equahty ((Tj) 
still holds for all u G D'"(G') and u G U^°°. Again, this implies that 
uu G 

The proof for the separate continuity of $y is similar to that of 
Lemma I3.3[ We need to prove that the map 

B'{G)xU-°^ C°°(G';f/), 

is separately continuous. This is the same as that the bilinear map 

D«(G')xf/-°^ ^ C{G]U), 

is separately continuous for all X G U(gc)- This is again true by 
checking that 

X{c^u) = qLx{io))u, w G D^(G), u e 

□ 

To summarize, we get a separately continuous bilinear map 
D'{G) X U-°° ^ f/°°, (w, u) ^ oou, 
which extends the action map 

Similarly, we have a separately continuous bilinear map 

D'{G) X V-^ V^, {u, v) ^ uv, 

which extends the action map 

D'{G)xV^V°°, {io,v)^iov. 

Now define the (distributional) matrix coefficient map by 

c : U-^ X ^ C-«(G), 

Cu^vi(^) ■= {(^u,v) = {u,u'^v), w G D'^(G'). 

The last equality is implied by ([8]) and the afore-mentioned separate 
continuity statements. 

Lemma 3.6. The matrix coefficient map a defined in p7]) is continu- 
ous. 
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Proof. Note that the spaces U~°°, V~°° and G^^(G) are all strong du- 
als of reflexive Frechet spaces. Therefore by |Tr67[ Theorem 41.1], it 
suffices to show that the map (ITT]) is separately continuous. First fix 
u G U~°°, then the map 

y-oo^C-«(G), v^c^^,, 

is continuous since it is the transpose of the continuous linear map 

Similarly, fix f G the map 

f/"- ^ c-«(G), u ^ Cu^,, 
is continuous since it is the transpose of the continuous linear map 

□ 

It is straightforward to check that fill I) extends the usual matrix 
coefficient map ([2]). The proof of the first assertion of Theorem 12.11 is 
now complete. 

To prove the second assertion of Theorem 12.11 we need two elemen- 
tary lemmas. 

Lemma 3.7. Let X : E ^ F be a G intertwining continuous linear 
map of representations of G. Assume that E is Frechet, smooth, and 
of moderate growth, and F is in the class J^H. Then E/Kei{X) is a 
representation of G in the class J^T-L, and X is a topological homomor- 
phism. 

As the proof is routine, we omit the details. 

Lemma 3.8. Let X : E ^ F be a continuous linear map of nuclear 
Frechet spaces. Equip the dual spaces E' and F' with the strong dual 
topologies. Then X is a topological homomorphism if and only if its 
transpose A* : F' — )■ E' is. When this is the case, both X and A* have 
closed images. 

Proof. The first assertion is a special case of |B87[ Section IV. 2, Theo- 
rem 1]. (Recall that every bounded set in a complete nuclear space is 
relatively compact.) 

Now assume that A is a topological homomorphism. Then as a Haus- 
dorff quotient of a Frechet space, £'/Ker(A) is complete, and so is Im(A), 
which implies that Im(A) is closed in F. By the Extension Theorem of 
continuous linear functionals, the image of A* consists of all elements 
in E' which vanish on Ker(A). This is closed in E' . □ 
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Recall that both V°°<^U°° and D'"(G') are nuclear Frechet spaces. In 
particular, they are both reflexive. The map c of (jl]) is the transpose 
of a G X G intertwining continuous linear map 

(Here we have used the canonical isomorphism E'®F' ~ (E^F)', for 
nuclear Frechet spaces E and F. See |Tr67[ Proposition 50.7].) Lemma 
13.71 for the group G x G implies that c* is a topological homomorphism. 
Lemma 13.81 then implies that c is a topological homomorphism with 
closed image. This completes the proof of the second assertion of The- 
orem [2TT1 

4. Proof of Theorem 12.31 

The argument is standard (c.f. |GK71] or |S74] ). We use the notation 
and the assumption of Theorem 12. 3[ As before, U~°° is the strong 
dual of V°°, and V~°° is the strong dual of U°°. Suppose that both 
Hom5'^([/°°, C^J and Hom5'2(V°°, C^J non-zero. Pick 

O^MoGHom5,(F°^,C^Jcf/-'" 

and 

0^t;oeHom5,(f/°°,C^JcV^-°°. 
Then the matrix coefficient c^Q^t,,, G C^^(G') satisfies the foUowings: 

it is an eigenvector of U(gc)*^, (by the irreducibility hypothesis) 

Cuo®vo{sx) = Xi{^) (^uo^voix) , s G 5*1, and 

By the assumption of the theorem, we have 

(12) C^O^JJQ ("^ ) Cug^t,g(x). 

Lemma 4.1. Let to G D''(G). Denote by 

n :D^(G) ^D^(G) 
the push forward map by r. Then 

uuq = if and only if (r^,(a;))^i;o = 0. 
Proof. As a consequence of ffT^ . we have 

(13) Cuof^voi^) = if and only if Cuo(iivo{T*{^^)) = 0. 
By the irreducibility of U°°, ojuq = if and only if 

{g{ujuo), vo) = for all g e G, 

i.e., 

{{Lgu)uo,Vo) = for all g E G. 
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By ([I3D, this is equivalent to saying that 

{{T^{Lgu))uo,Vo) = for all 5f G G. 

Now the lemma follows from the following elementary identity and the 
irreducibility of V°°: 

□ 

End of proof of Theorem \2.3[ Let 

0^M[,GHom5,(r~,C^Jc[/-°° 
be another element. Applying Lemma 14.11 twice, we get that for all 

uuq = if and only if lou'q = 0. 

Therefore the two continuous G homomorphisms 

$ : a; (-7- ujuq, and $' : t-)- uju'q, 

from D''{G) to U°° have the same kernel, say J. Here and as before, 
we view D''{G) as a representation of G via left translations. Both $ 
and $' induce nonzero G homomorphisms into the irreducible Harish- 
Chandra representation f/°° 

: D'^(G)/J-> f/°°, 

without kernel. Lemma [3.71 says that D'"(G)/J is a representation of 
G in the class J^H. By Schur's lemma for the Category $' is a 
scalar multiple of $, which implies that Uq is a scalar multiple of uq. 
This proves that 

dimHom5,(l^°°,C;,J = 1- 

Similarly, 

dimHom5,(f/°°,CxJ = L 

To finish the proof of Part (a) of Theorem 12.31 we first note the 
identity Cuo^vo{^~^) = Cvo(Siuo{^) (a-s generalized functions). The equal- 
ity of matrix coefficients in fll2p will then imply that U?° and V°° are 
equivalent, in view of Remark 12.21 Part (B). 

Part (b) of Theorem 12.31 is immediate as the matrix coefficient c^q^dq 
would have to be zero if there were nonzero uq G Roms^iV^^ , C^^) and 
nonzero Wo G Homgj ([/°°, J. □ 
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